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ABSTRACT 

A  complex  robotics  system  using  vision,  tactile  sensors  and  external  tools 
is  used  to  assemble  objects  of  variable  shape  (jig-saw   puzzle  pieces)   and  orientation. 
Control  of  the  process  is  shared  between  two  computers,  one  used  for  vision  and  the 
other  for  robot  control.    Partial  results  are  presented.    This  is  a  thesis  proposal. 
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1.   Introduction 

Present  industrial  robots  can  perform  repetitive  work  with  great  accuracy 
and  speed  on  a  continuous  basis  [1].  However,  despite  these  advantages, 
these  robots  are  very  Hmited  in  the  type  of  work  that  they  can  perform  [2]. 
Their  load/weight  ratio  is  very  poor,  being  about  an  order  of  magnitude  less 
than  that  of  a  worker.  Their  ability  to  do  batch  jobs  is  limited  by  a  lack  of 
good,  inexpensive,  commercially  available  end  effectors  and  sensory  equip- 
ment, both  of  which  are  needed  to  perform  assembly  work  that  employs 
adaptive  control.  In  addition,  present  vision  systems  are  too  slow  for  real- 
time robot  control,  due  in  part  to  the  enormous  amount  of  computation 
required  to  process  an  image.  This  limits  industrial  application  [3],  but  the 
situation  is  improving  due  to  new  super  fast  chips,  parallel  processing  used  in 
dedicated  hardware,  and  the  development  of  new  general  purpose  vision  sys- 
tems. 


-  Work  on  this  project  has  been  supported  by  Office  of  Naval  Research  Grant  N00014- 
82-K-0381,  National  Science  Foundation  CER  grant  DCR-83-20085,  and  by  grants  from  Digi- 
tal Equipment  Corporation  and  IBM  Corporation. 
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2.   The  purpose  of  this  project 

The  main  aim  of  this  project  is  to  implement  and  demonstrate  a  capabil- 
ity to  assemble  variable  and  complex  shaped  parts  with  relatively  small  toler- 
ances. This  is  a  substantially  more  demanding  task  than  the  "peg  in  a  hole" 
problem  treated  in  the  literature  [4], [5],  due  to  the  complex  contour  of  the 
parts  to  be  assembled,  which  are  wooden  jig-saw  puzzle  pieces. 

Previous  work  by  Edith  Schonberg  and  Haim  Wolfson  at  NYU  Robotics 
Laboratory  developed  algorithms  for  simulated  jig-saw  puzzle  assembly  using 
a  vision  system.  In  their  work,  the  puzzle  pieces  are  initially  digitized  via  a 
vision  system,  and  a  puzzle  solution  is  obtained.  This  solution  is  stored  in 
memory  and  can  be  displayed  graphically  (see  Fig.  1,  Assembled  jig-saw  puz- 
zle). However,  due  to  small  vision  system  errors  and  to  the  fact  that  only  the 
2-D  contour  of  the  puzzle  pieces  is  used  (and  not  the  image  actually  present 
on  the  puzzle),  frequent  small  position  errors  occur  in  the  solution. 

This  thesis  will  study  the  problems  of  actual  physical  assembly  of  jig-saw 
puzzles.  The  small  errors  which  Schonberg  and  Wolfson  could  ignore  are 
expected  to  prove  central  to  our  work.  We  propose  to  use  puzzle  pieces 
because  they  pose  a  significant  challenge  to  mechanical  assembly.  Most  of 
the  difficulties  already  arise  in  attempting  an  automatic  assembly  of  two  puz- 
zle pieces.  Hence  the  first  part  of  our  effort  will  concentrate  on  development 
of  techniques  which  hand  this  case  well,  then  work  will  extend  to  the  whole 
puzzle.  In  dealing  with  whole  puzzles,  the  physical  assembly  procedures  to  be 


developed  for  this  thesis  may  also  have  to  cope  with  gross  errors  (since  the 
Schonberg-Wolfson  work  will  sometimes  give  several  solutions  which  is 
imable  to  disanbiguate,  so  that  the  only  way  to  validate  may  be  through 
actual  assembly  [6]). 

3.   Installation 

The  assembly  system  to  be  developed  will  consist  of  a  7565  IBM  robot,  a 
vision  system,  tactile  sensors,  and  an  assembly  tool.  The  design  of  this  special 
tool  and  sensors  constitutes  a  significant  part  of  this  project  (see  Fig.  2  which 
shows  our  manipulator  system  with  vision  equipment  mounted,  also  Fig.  3  - 
Fine  motion  assembly  installation).  An  existing  vision  system  will  be 
employed.  This  consists  of  a  Fairchild  CCD  4001  Robotics  Camera,  with  a 
256x256  sensor  array,  and  a  VICOM  image  processor  system,  which  manipu- 
lates a  512x512  array.  The  Fairchild  camera  allows  the  sensor  head  to  be 
placed  on  the  robot  arm,  while  the  rest  of  the  camera  remains  outside  the 
work  envelope.  This  will  simplify  installation  and  reduce  the  risk  of  damage 
to  sensitive  electronics  in  the  main  part  of  the  camera.  The  camera  will  send 
the  analogue  image  of  a  puzzle  piece  to  the  VICOM  which  digitizes  it  and 
displays  the  boundary  and  the  corners  of  the  puzzle  piece  on  a  color  monitor. 
A  MICRO  VAX  host  computer  will  control  the  whole  vision  process.  Data 
obtained  from  the  host  computer  will  be  sent  to  a  SERIES  1  minicomputer 
which  controls  the  robot  arm  (IBM  7565).    The  SERIES  1  will  also  control 
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the  operation  of  a  special   assembly  tool  used  to  handle  the  pieces.   This  tool 
is  discussed  in  more  detail  in  section  5. 

4.  Assembly-tool  design  considerations 

The  shape  of  the  puzzle  pieces  and  the  characteristics  of  the  7565  gripper 
require  a  special  tool  to  perform  the  assembly.  This  tool  will  use  a  vacuum 
cup,  which  has  the  advantage  that  contact  would  only  occur  at  the  top  of  the 
piece,  minimizing  the  complexity  of  the  operation.  This  approach  is  also  use- 
ful because  during  the  final  stage  of  assembly,  vacuum  can  be  replaced  by 
short  pressure  bursts  to  help  force  the  piece  into  place  [10].  The  upper  end 
of  the  tool  will  have  a  square  shape  to  allow  a  flat  contact  surface  with  the 
gripper.  A  small  hole  (0.05"  diameter)  will  help  align  the  tool  with  the 
gripper,  using  the  gripper's  Light  Emitting  Diode  (LED)  sensor. 
Vacuum/pressure  pipes  and  electrical  circuitry  for  the  tool  will  be  routed 
through  the  robot  vertical  arm  to  clear  the  workspace  of  any  unwanted  exter- 
nal objects.  These  considerations  led  to  the  (tentative)  design  for  the  assem- 
bly tool  presented  in  Fig.  7.  Additional  experiments  will  be  necessary  to 
determine  the  sensory  requirements  for  assembly  verification. 

5.  Fine  assembly  strategy 

Once  the  tool  is  secured,  the  tip  of  the  tool  with  its  suction  cup  will 
approach  the  piece  to  be  picked  and  descend  along  a  vertical  path.  Motion 
will  stop  when  the  sum  of  the  force  readings  on  the  tip  sensors  of  the  gripper 
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exceeds  a  certain  threshold.  Once  the  tool  is  on  top  of  the  puzzle  piece,  the 
robots'  force  sensors  will  cause  the  robot  controller  to  activate  an  external 
relay  which  in  turn  would  start  a  vacuum  pump.  The  air  pressure  sensor  on 
the  tool  would  then  indicate  via  the  robot  controller  when  a  desired  vacuum 
is  achieved  which  means  that  the  puzzle  piece  is  secured.  The  piece  is  then 
moved  to  the  location  of  fine  assembly  with  the  necessary  (raw)  translation 
and  rotation  data  previously  obtained  by  the  vision  system  (as  presented  in 
Chapter  6).  In  the  actual  assembly,  the  piece  that  is  to  be  inserted  will  ini- 
tially be  inclined  at  about  15  degrees  to  the  surface  of  the  work  table  and 
pushed  over  the  surface  of  the  matching  puzzle  piece.  Once  the  pieces  to  be 
assembled  are  in  proper  initial  contact,  with  the  compliance  of  the  cup 
exercising  a  force  on  the  horizontal  puzzle  piece  (see  Fig.  7),  forces  and 
moments  will  be  measured  by  the  force  sensors  on  the  gripper.  WTien  these 
readings  decrease  due  to  a  hole  in  the  horizontal  piece,  the  piece  that  is  held 
by  the  tool  can  be  lowered  into  place  by  reducing  the  angle  of  inclination 
and/or  stopping  the  vacuum  pump.  The  last  step  associated  with  the  assem- 
bly is  to  verify  that  the  assembly  is  correct.  This  will  be  done  by  vision  and 
force  sensing.  It  is  important  to  note  that  the  range  of  the  force  sensors  on 
the  gripper  is  0  to  100  Newtons,  with  one  bit  noise,  which  in  force  terms  is 
equal  to  0.15  Newtons  (see  Fig.  6  Force  sensor  indications  with  0.15  New- 
tons  random  "jumps").  This  noise  therefore  requires  that  the  threshold  values 
be  lowered  by  0.30  Newtons  (for  the  worst  case)  and/or  averaged  over 
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several  consecutive  readings.  The  range  of  forces  used  for  fine  assembly  is 
5-6  Newtons,  which  implies  that  the  noise  uncertainty  is  approximately  6%. 
The  speed  with  which  fine  assembly  is  done  will  have  to  be  quite  small.  In 
order  to  avoid  vibrations  or  the  loss  of  a  picked  piece,  the  speed  would  have 
to  be  of  the  order  of  5%  of  the  nominal  speed  of  the  7565  robot. 

6.  Vision  System  software 

An  initial  stage  of  the  software  will  ascertain  the  position  of  each  puzzle 
piece  (see  Fig.  4  -  Data  acquisition  program  called  from  7565  controller  and 
running  on  the  host  computer).  This  will  be  done  by  moving  the  camera  in  a 
search  pattern.  During  the  search,  the  camera  will  remain  at  a  fixed  distance 
above  the  work  table,  allowing  the  use  of  a  fixed  focal  length  lens  and  ensur- 
ing a  sharp  image.  Detected  pieces  will  be  analyzed  using  the  existing  SHAPE 
program,  whose  output  is  a  sampled  curve.  This  information  will  then  be 
used  in  the  existing  MATCH  program  to  determine  matching  sides  for 
corresponding  puzzle  pieces.  This  program  will  output  the  rotation  necessary 
to  align  the  two  matching  sides. 

7.  Communications 

The  host  computer  that  coordinates  the  vision  system  and  obtains  the 
simulated  assembly  will  run  in  an  UNIX  environment.  This  is  a  different 
operating  system  from  that  of  SERIES  1  minicomputer  (7565  controller),  and 
communication  between  the  two  systems  creates  problems.   To  cope  with  this 


situation  a  7565  Asynchronous  Communications  Support  PRPQ  program  will 
be  installed  on  the  robot  controller  [II].  All  communications  will  be  done  on 
a  RS232  port  available  on  SERIES  1,  with  data  in  ASCII  format.  The  PRPQ 
program  is  not  supported  by  IBM  and  it  has  to  be  "customized"  for  the  needs 
of  this  project.  All  asynchronous  communication  techniques  for  interlacing 
the  data  transfers  into  a  unique  global  program  running  on  two  different 
computers  are  still  to  be  developed. 
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Figure  1.  Assembled  jig-saw  puzzle 
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FIGURE  2  MANIPULATOR  ASSEMBLY     WITH   VISION    EQUIPMENT   MOUNTED   ON 
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Fig.  4  Data  aquisition  program  called  from  75f)5  controler  and  running  on 
host  computer. 
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Fig.  5  Rotation  and  translation 
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